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Abstract 



A model is proposed for optimizing simultaneously combinations of test-based 
decisions using Bayesian theory. The decision problem addressed consists of a 
selection, a placement, and a mastery decision. Combinations of such decisions 
can be found, for instance, in computerized adaptive instruction networks. 
Compared with separate optimization, a simultaneous approach has two 
advantages. First, test scores used in previous decisions can be used as "prior 
data" in later decisions and the efficiency 01 the decisions can be increased. 
Second, more realistic utility structures can be defined using final success criteria 
in utility functions for earlier decisions. An important distinction is made between 
weak and strong decision rules. As opposed to strong rules, weak rules are 
allowed to be a function of prior test scores. Conditions for optimal rules to be 
monotone are presented. Also, it will be shown that the optimal weak monotone 
rules are compensatory by nature. Results from an empirical example of 
instructional decision making will be presented to illustrate the differences 
between a simultaneous and a separate approach. 
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Introduction 

Decision problems in educational and psychological testing can be classified in 
many ways. In van der Linden (1990) the following four types of test-based 
decisions are distinguished: selection, mastery, placement, and classification. 
Typically, modem instructional systems as individualized study systems (ISS's), 
mastery learning, and computer-aided instruction (CAI) do not involve one single 
decision but can be conceived of as networks of nodes at which one of the types 
of decisions above has to be made (van der Linden, 1990; Vos, 1990, 1991, 
1993, 1994a; Vos & van der Linden, 1987). 

The question is raised how such networks of decisions should be 
optimized. An obvious approach is to address each decision separately, 
optimizing its decision rule on the basis of test data exclusively gathered for this 
individual decision. This approach is common in current design of instructional 
systems. The purpose of this paper is to show that multiple decisions in networks 
can also be optimized simultaneously using Bayesian theory (e.g., DeGroot, 1970; 
Ferguson, 1967; Lindgren, 1976). The advantages of a simultaneous approach are 
twofold. First, data gathered earlier in the network can be used to optimize later 
decisions. The use of such prior information can be expected to enhance the 
quality of the decisions - in particular if only small tests or sets of 
multiple-choice items are administered at the individual decision points. As a 
result of the more efficient use of all data available in the network of 
simultaneous decisions, one might expect that the expected utility for a 
simultaneous approach is larger than for a separate approach. Second, a more 
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realistic definition of utility or loss functions is possible, since these functions 
can now be defined on the ultimate success criterion instead of on intermediate 
criteria measuring the success on individual treatments. 

In this paper, a decision network consisting of a selection, a placement 
(with two treatments), and a mastery decision will be used to make our point. 
First, the selection-placement-mastery problem will be formalized. Then 
important distictions will be made between weak and strong as well as monotone 
and nonmonotone decision rules. Next, it will be indicated under which 
conditions optimal rules take a monotone form. Finally, results from an empirical 
example of instructional decision making will be presented to illustrate the 
differences between a simultaneous and a separate approach. 

The Selection-Placement-Mastery Problem 

A flowchart of the selection-placement-mastery problem is given in Figure 1. 
Real-life systems usually have more decision points. 



Inset t Figure 1 about here 



The first decision to be made is a selection decision. A selection decision is made 
when a test is administered before a treaunent takes place, and only students 
promising satisfactory results on the criterion are accepted for the treaunent For 
instance, the treaunent may be a remedial module in which some prerequisite 
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knowledge is offered in preparation of the treatments to come later in the 
instructional program. After the initial treatment a placement decision follows, 
where St 'tents are assigned to one of two possible treatments based on their 
placement scores. With placement decisions the success of each of the treatments 
is measured by the same criterion. The paradigm underlying placement decisions 
is the Aptitude Treatment Interaction (ATI) hypothesis from instructional 
psychology, which assiunes that students may react differentially to instructional 
treatments, and, therefore, that different treatments may be best for different 
students. In general, students with high test scores on the placement test will be 
assigned to treatment 1. Therefore we will sometimes refer to treatment 1 and 0 
as the 'higher' and Mower' treatment, respectively. Finally, on the basis of the 
mastery test, it is decided whether the student has mastered the subject matter in 
the treatment sufficiently and may proceed with the next treaunem. Students who 
fail, however, have to relearn the material offered in the treatment and prepare 
themselves for a new mastery test. 

It is important to realize that, although the nature of the decisions shown 
in Figure 1 is sequential, the decision rules are optimized simultaneously. Also, 
the data used to optimize the rules is supposed to come from the following 
statistical experiment: First, students exposed to the same selection test are 
randomly drawn without reference to the score on the selection test in question 
and accepted for the initial treaunent. Next, the accepted students are randomly 
assigned to either treatment 0 or 1, after which their performances on the mastery 
test are measured. 

In the following, we shall assume that for a randomly sampled 
individual the observed-score selection test variable X, the observed-score 
placement test variable Y, the observed-score mastery test variable Z, and the 



Selection-Placement-Mastery Decisions 

5 



classical test theory true score variable T underlying Z, i.e., the criterion common 
to the treatments j (j = 0,1), are continuous raiidom variables. Furthermore, it will 
be assumed that the relation between X, Y, Z, and T can be represented by a 
density function tj(x,y,z,t). Since the treatments take place between the placement 
and the mastery test, this relation may entail different density functions for each 
treatment. Therefore, fj(x,y,z,t) is indexed by j. However, since both the selection 
and placement test are administered prior to the treaunent j, the density functions 
q(x), s(y), and h(ylx) of respectively X, Y, and Y given X = x will not be 
indexed by j. 

Treaunent-denendcnt Mastery Rules in a Simult aneous Approach 
In the case of a simultaneous approach to optimizing decision networks, an 
important distinction is made with respect to decision rules on the mastery test 
for students assigned to treaunents 0 and 1. These rules will be different to allow 
for the "collateral information" present in the fact that examinees have followed 
different previous treaunents before they take the mastery test. 

Weak Monotone and Strong Monotone Rules 

A decision rule specifies for each possible realization (x.y.z) of the sample space 
XxYxZ which action has to be taken. Here, we only consider monotone rules; 
that is, rules using cutting scores. Let x c , y c , z cj , and t c denote the cutting scores 
on X, Y, Z for students assigned to treaunent j, and T, respectively, where t c is 
set in advance by the decision-maker (j = 0,1). Clearly, the 
selecuon-placement-mastery problem now consists of simultaneously setting 
cutting scores x c , y c , and z Q} that, given the value of t c , maximize the expected 
utility. 
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In general, when setting y c as well as z c j, prior achievement on the 
selection test can or can not be taken into account. In addition, the observed 
scores on the placement test may or may not influence the cutting scores to be 
set on the mastery test. Intuitively, students with selection scores equal to or just 
above x c must compensate their relatively low selection scores with higher scores 
on both the placement and mastery test Reversely, it seems reasonable that 
students with placement scores far above y c should be advanced earlier than 
students with placement scores equal to or just above y c . 

To distinguish between cases where prior achievement has or has not to 
be taken into account, those rules will be called weak monotone and strong 
monotone rules, respectively. For each x > x c , the weak cutting score on the 
placement test is defined as a function y c (x). Similarly, for each x £ x c and y, 
the weak cutting score on the mastery test under treatment j is given as a 
function z oi (x,y). For strong monotone rules, however, both strong cutting scores 
y r and z r : are set independently of observed test scores on prior tests. Since the 
most general fonn of the decision rule is a weak simultaneous rule, this type of 
rule will be treated first. Later on, strong simultaneous rules will be considered as 
a special fonn of weak simultaneous rules. 

Each action will be denoted by a^ (i,j,k = 0,1), where i = 0 or 1 stands 
for rejecting or accepting a student, j = 0 or 1 stands for assigning an accepted 
student to treatment 0 or 1, and k = 0 or 1 stands for retaining or advancing an 
accepted student. Since for a rejected student no further placement and mastery 
decisions are made, the indices j and k will be dropped for i = 0. 

For the decision network of Figure 1 a weak simultaneous rule 8 can be 
defined as: 
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{(x,y,z) : 5(x,y,z) = aQ} = AxRxR 

((x t y,z) : 5(x,y,z) = a 100 } = A C x B(x) x D Q (x,y) 

{(x,y,z) : 5(x,y,z) = a 10 ,} - A C x B(x) x D Q C (x,y) (D 

{(x,y t z) : 6(x,y,z) = a n0 } = A C x B C (x) x D^x.y) 

{(x,y,z) : 8(x,y,z) = a m ) = A C x B C (x) x Dj^x.y), 

where A, A C , B(x), B C (x), Dj(x,y), and Dj C (x,y) stand, respectively, for the sets 
of x, y, and z values for which a student is rejected or admitted for the initial 
treatment, an accepted student is assigned to treatment 0 or 1, and an accepted 
student is retained or advanced under treatment j (j a 0,1). R stands for the set of 
real numbers. Thus, a weak monotone rule 8 can be defined for our example as 
follows: 

if X < x c , y e R, and ZeR 
if X > x c , Y < y c (x), and Z < z cQ (x,y) 
if X > x c , Y < y c (x), and Z > z c0 (x,y) (2) 
if X > x c Y > y c (x), and Z < z cl (x,y) 
if X > x c , Y > y c (x), and Z > z cl (x,y). 

Since we confine ourselves to monotone rules in this paper, following 
common practice in criterion-referenced testing, we are to show that there are no 
nonmonotone rules with larger expected utility, or, equivalently, that the subclass 
of monotone rules constitutes an essentially complete class (e.g., Ferguson, 1967; 
Karlin & Rubin, 1956). Conditions under which the subclass of monotone rules is 
essentially complete will be given later on. If these conditions are satisfied, a 
monotone solution is said to exist. 



8(X,Y,Z) = ( 



a 100' 
a 101> 
a 110» 
a lll- 
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Other Types of Rules for the Comhired Decision Problem 
The introduction of weak monotone and strong monotone rules implies that 
optimal rules can be considered which take both a weak monotone and strong 
monotone form. Since strong monotone rules are special cases of weak monotone 
rules, this type of rules can only be optimal if along with the conditions under 
which weak monotone rules are optimal certain additional restrictions are met. 

Furthermore, since in educational testing one is accustomed to using 
strong cutting scores, rules with maximum expected utility in the subclass of 
strong monotone rules can also be calculated without bothering about 
monotonicity conditions. To stipulate the difference with the optimal (strong 
monotone) rules, this type of rules will be termed Strong Monotone Rules with 
Maximum Expected Utility (SMMEU) rules. It should be emphasized that an 
optimal rule from the set of all possible simultaneous rules only takes a strong 
monotone form if the additional (rather strict) restrictions are met. 

An Additive Utility Structure for the Combined Decision Problem 

Th utility structure of the combined decision problem, Uy k (t), is supposed to be 

an additive function of the following form: 



lire nonnegative weights. For a rejected student, zero utilities for the separate 
placement and mastery decisions are assumed. Hence, it follows from (3) that 



u ijk (t) = w lU] 



ij (s) (0 + w 2 UjW(t) + w 3 u k (m) (t), 



(3) 



where Uj^(t), Uj^\t), and u^ in \t) represent, respectively, the utility functions 
for the separate selection, placement, and mastery decisions, and wj, w 2 , and w^ 




j (s) (t) for all j and k. 
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Since the utility functions u^ s \t), Uj<P>(t), and u k ^ m \t) are allowed to 
assume different forms, Equation 3 offers us a great deal of flexibility to describe 
utility structures in practical applications. Furthermore, since utility is supposed to 
be measured on an interval scale, the weights in (3) can be rescaled as follows: 

u ijk (t) = w lUi (s) (t) = w 2 u j ( P ) (t) + (l-w r w 2 )u k (m) (t), (4) 

with 0 < Wj, w 2 , (l-wj-w 2 ) ^ 1. 

As mentioned earlier in the Introduction, one of the main advantages of 
a simultaneous approach is that more realistic utility structures can be defined. 
Equations 3-4 demonstrate this fact nicely, since a utility function defined on the 
ultimate criterion of the decision network is formulated not only for the mastery 
decision but also both for the selection and placement decision. This choice is in 
line with the philosophy underlying ISS's. 

Expected Utility in a Simultaneous Approach 

It is important to realize that the expected utility in a simultaneous approach 
according to the statistical experiment described earlier is composed of eight 
terms, whereas according to (1) only five possible actions can be identified. The 
reason is that action a^ contributes four terms to the expected utility, namely one 
for each possible combination of outcomes of the decisions that follow. The 
placement rules in the calculation of this part of the expected utility are different 
for students rejected or accepted for the initial treaunent to allow for this 
"collateral information". The mastery rules are not only treaunent-dependent but 
also depend on the information if students are rejected or accepted for the initial 
treaunent. 
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For the most general form of the decision rules according to (1), the 
expected utility in a simultaneous approach can be calculated as follows: 

E[U sim (A C ,B C (x),B R C (x), D 0 C (x,y), D^x), D rf) C (x,y),D rl C (x,y))] = 
Ja i r (x) Jb^x.y) Jr "O00(t)f0(^y.^t)dtdzdydx ♦ 
L JB r (x) 4 C (x y) Jr uooi(t)f 0 (x.y. Z .t)dtdzdydx ♦ 

Ja 4 r c (X ) ki^y) Jr "oiowfic.y.^odtdzdydx ♦ 

Ja 4 f C (x) bfay) Jr "Oll^f^x.y.z.tldtdzdydx + 

Ja c Jb( X ) JD 0 (x.y) J"r u 100(0f 0 (x.y. Z .t)dtdzdydx ♦ (5) 

Ja c Jb ( x) SoC m Jr miWu****** ♦ 

J A C Jb C (x) JDi(x t y) Jr »U(W*W)dtdzd y dx + . 
J A C Jb C (x) 4 C (x y) J R "1 1 i(t)f,(x.y.7.,t)dtdzdydx . 

where B r (x), B r C (x), Djj(x,y), and D^U.y) represent, respectively, the sets of y 
and z values for which a rejected student is assigned to treatment 0 or 1 and 
failed or passed the mastery test under treatment j (j = 0,1). 

Taking expectations, completing integrals, rearranging terms, and using 



14 



Selecticm-Placemcnt-Mastery Decisions 

11 



u 0jk^ = w l u 0 ^ ^ or a " J m ^ ^ ^ can ^ w " tten m posterior form as 

E[l! sim (A C > B C (x),B r C (x),D 0 C (x,y),D 1 C (x,y))) « w^luJV)] ♦ 
4 C x w 1 {E 1 [uJ S) (T)!x,y]-E 0 [uJ S) (T)|x,y]}h(y|x)q(x)dydx ♦ 



J A c| E ot u ioo( T )- w iV( T )l x i + 

(6) 

JrLc E 0 [u 101 (T)-u 1 o 0 (T)|x,y,z]in 0 (z|x,y)h(y|x)dzdy ♦ 
l 

r f 

J^(^ ^Ej[uiji(T)-UijQ(T)|x,y t z]mj(z|x t y)dz}h(y|x)dy|q(x)dx t 

with nij(zlx,y) being the p.d.f. of Z given X = x and Y = y under treatment j and 
where EjU indicates that the expectation has been taken over a distribution 
indexed by j. 



Sufficient Conditions for Monotone Solutions 



In this section, sufficient conditions for optima simultaneous rules to be 
monotone will be derived. First, conditions under which optimal rules take a 
weak monotone form will be derived. Next, it will be indicated how sufficient 
conditions for strong monotone solutions can be obtained from the previous case 
by imposing certain additional restrictions on fj(x t y,z,t). 
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Sufficient Conditions for Weak Monotonicity 

To derive the conditions under which optimal rules take a weak monotone form 
for our example, the following well-known theorem (e.g., Ferguson, 1967, p. 
201) is needed: 

For every function f(x) with J* |f(x)|dx < «> 
and any set S of x values, 

it holds that J s f(x)dx < £ f(*) d * with S Q = {x: f(x) > 0}. (7) 

Applying this theorem first to the inside integrals w.r.t. z, next to the 
middle integrals w.r.t. y, and finally to the outside integral w.r.t. x in (6), it can 
be verified that an upper bouno to the expected utility is obtained for: 

D * 0 C (x,yMz: E 0 [u 10 i(T)-ui 0 o(T)|x,y,z) > 0), (8) 
D*j C (x,y)=fc: E 1 [u 111 (T)-Ui 1 o(T)|x,y,z] > 0), (9) 

B* C (x)H{ y: £ (2j-l){Ej[u lj0 (T)|x,y] ♦ 
j=0 

JN C (x Ejtu 1 j 1 (T)-u 1 jo(T)|x,y,z]mj(z|x,y)dz} > 0), (10) 
B * J\x) b { y: E,luJ s) (T)| x ,y] - EoluJ'Voix.yl > 0), (1 1) 
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A* L ^ {x: E 0 [u 100 (T)-.W lU 5 s; (T)tx] + 

JrJd* c ^ Ju ioi (T >~ u ioo^l x *y' z i m o (z l x -y> h( y l x)dzd y + 

{ 0 (12) 

L *C E j [u ljl (T)-U lj0 (T)|x,y t z]m j (z|x t y)dz}h(y|x)dy > 0). 
j'(x,y) 

We are now able to specify conditions for weak monotonicity. For weak 

monotone rules, the sets D*j^(x,y), B*^(x), and A*^ take the form [z c j(x,y),oo) t 

[y c (x),°*), and [x c ,<»), respectively. As opposed to these sets, however, it is 

assumed that the sets B ^ (x) take the form (-»,y cr (x)]. The reason for assuming 

a somewhat different form for the sets B r v (x) is that under quite realistic 

conditions on the test score distributions and utility functions (Vos, 1994b), a 

* n 

weak monotone solution can be guaranteed. The form of B ^"(x) suited above 

implies that rejected students, unlike accepted students, with low scores on the 

placement test will generally be assigned to the 'higher' treatment. 

It follows that the optimal weak rules take a monotone form if the 

left-hand sides of the inequalities in (8)-(9), (10), and (12) are increasing 

functions in z for all x and y, in y for all x and z, and in x for all y and z, 

respectively, whereas the left-hand side of the inequality in (11) is required to be 

decreasing in y for all x. In trie empirical example below it will be examined if 

these conditions hold. 

The sets l)* 0 C (x,y), D* l C (x,y), B* r C (x), B* C (x), and A* C can be 

* n 

obtained in the following way: For each x and y, first the sets D q (x,y) and 
D i (x,y) can be computed from (8) and (9), respectively, whereas for each x 
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the sets B* f ^(x) can be computed from (11). Then, for each x, inserting 
D*j^(x,y) into (10), the sets B*^(x) can be computed. Finally, inserting 
D*£(x,y) and B*^(x) into (12), the set A*^ can be computed. 

Monotonicitv Conditions for Strong Simultaneous Rules 

For strong simultaneous rules, B^(x) and B f ^(x) are not allowed to depend on x 
and Dj C (x,y) not on x and y. Let vj(t|x,y,z) and rj(t|x,y) denote the p.d.f.'s of 
T given X = x, Y = y, and Z = z and T given X = x and Y = y under treatment 
j, respectively (j = 0,1). In addition to the inonotonicity conditions for weak 
simultaneous rules, it then follows from (8)-(U) that an upper bound to the 
expected utility is reached for a strong monotone rule if vj(t|x,y,z) does not 
depend on X = x and Y = y, whereas rj(t | x,y) and mj(z | x,y) do not depend on 
X = x(j = 0,l). 



Calculation of Simultaneous Rules 



In this section, it will be indicated how the different types of rules discussed 
earlier can be calculated. First, it will be shown how optimal rules can be 
obtained being both of a weak monotone and strong monotone form. Next, it will 
be indicated how SMMEII rules can be derived by maximizing the expected 
utility over the subclass of strong monotone rules. 

Optima] Simultaneous Rules 

Assuming the conditions for weak inonotonicity are satisfied, optimal weak 
cutting scores can now be obtained for those values of z c j(x,y), y c (x), y cr (x), and 
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x c for which the inequalities in (8)-(12) turn into equalities. The optimal weak 
monotone rule is then given by x c , {y c (x)lx £ x c ), {y cr (x)lx < x c ), {z c Q(x,y)lx > 
x c , y < y c (x)}, and {z c j(x,y)lx > x c , y > y c (x)}. Assuming the additional 
restrictions on fj(x,y,z,t) are also satisfied, optimal strong cutting scores can be 
obtained by solving the resulting system of equations simultaneously for x c , y c , 
y cr z c0 , and z cl . 

SMMEU Rules 

* * * * * 
The set of SMMEU cutting scores, say x c , y c , y cr z c q, and z cl , can be 

computed straightforward by inserting A^' = [x c ,oo), B^(x) = ly c ,°°), B f ^(x) = 

[y cr *>), and Dj^(x.y) = lz C j,«) into (6), differentiating w.r.t. x c , y c , y cr z c q, and 

z c j, setting the resulting expressions equal to zero, and solving simultaneously 

for x c ,y c , y cr z c0 , and z Ql 

Since no analytical solutions for these systems of equations could be 

found, the cutting scores can be calculated via a numerical approximation 

procedure such as Newton's iterative algorithm for solving nonlinear equations. 

For the present systems of equations, this algorithm was implemented in a 

computer program called NEWTON. Another prognun, UTILITY, was written to 

analyze differences in expected utility for the various rules. Copies of the 

programs are available from the author of the paper upon request. 



Optimal Separate Rules 

It is observed that optimal rules for the separate decisions can easily be found by 
imposing certain restrictions on the expected utility in a simultaneous approach. 
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First, for the separate selection decision it holds that mQ(zlx,y) = 
mj(zlx,y) » m(zlx,y) and v*2 55 (l-w^-t^) = 0, since both treatments coincide and 
there are zero placement and mastery utilities in this case, respectively. 
Substituting these restrictions into (6), the expected utility for the separate 
selection decision, EHJ^p^A^)], becomes 

E[u£p(A C )] =E[uJ S) (T)] ♦ J^ c E[uj s) (T)-uJ S) (T)|x]q(x)dx. (13) 

Next, for the septate placement decision it holds that = R and Wj = 
(l-wj-W2) = 0, since all students are accepted for the initial treatment and there 
are zero selection and mastery utilities in this case, respectively. Substitution of 
these restrictions as well as B v (x) = into (6) results for the expected utility 
of the separate placement decision, EfU™ 

(p) (B C 

E f u£>(B C )] = 

EoluJV)] ♦ £ c {E 1 [u ( 1 p) (T)|yl-E 0 [uJ p) (T)|y]}s(y)dy. (14) 

Finally, for the separate mastery decision it holds that A = R, 

mQ(zlx,y) = mj(zlx,y) = m(zlx,y), and wj = W2 = 0, since all students are 

accepted for the initital treatment, both treaunents coincide, and there are zero 

selection and placement utilities in this case, respectively. Inserting these 

C r 

restrictions as well as Dj (x,y) = 0 into (6), the expected utility for the 
separate mastery decision. I:[W^p\o^)], becomes 

E[u£jJ<» C )l = EIuJ m) (T)] + £ c E[uj in) (T)-uJ ,n) (T)|z]p(z)dz, (15) 
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where p(z) denotes the marginal distribution of Z. 

Analogous to a simultaneous approach, it can easily be verified that 
upper bounds to E[U^(A C )], E[U*J(B C )h and EfU^D 0 )] are obtained 
for the sets of x, y and z values for which the integrands in (13), (14), and (15) 
are nonnegative, respectively. For monotone rules, these sets take the form 
[x c ,°°)» [y c »°°)» and [z c ,«>), respectively. Assuming the monotonicity conditions 
for the separate decisions are satisfied, it follows that optimal cutting scores for 
the separate selection, placement, and mastery decisions, say * c , y c » and z c , 
can be obtained by solving the integrands in (13), (14), and (15) for x c , y c , and 
z c , respectively. For further details, see Mellenbergh and van der Linden (1981), 
van der Linden (1981), and van der Linden and Mellenbergh (1977). 



An Empirical Example 



The procedures lor computing the optimal rules were applied to a sample of 71 
freshmen studying medicine. Treatments 0 and 1 consisted of an interactive video 
(IV) and a computer-aided instructional (CAI) program. Ordering the treatments 
in this way was motivated by the fact that the IV-program contained more 
examples and exercises than the CAI-prognun, implying that students with high 
scores on the placement test were generally assigned to treatment 1. 

The selection, placement, and mastery tests consisted each of 25 
free-response items with test scores ranging from 0 to 100. It should be 
emphasized that the example given in this paper was used only to illustrate the 
models. The use of small samples is generally not recommended, because the 
estimated model parameters may have large errors which tend to propagate when 



21 



Selection-Placemenl-Mastery Decisions 

18 



computing optimal rules. 

The teachers of the course considered a student as having mastered the 
subject matter if he/she could answer correctly at least 60% of the total domain 
of items. Therefore t c was fixed at 60. 

Multivariate Normal Distribution 

It was assumed that the variables X, Y, Z, and T followed (possibly) different 
multivariate normal distributions under both treatments. Under this assumption, 
the means, variances, correlations, and reliability coefficients of the mastery test 
scores were estimated using maximum likelihood estimates and coefficient alpha, 
respectively. Table 1 shows the results of the computations. 



Insert Table 1 about here 



When applying the procedures in this paper, it should always be checked 
whether the assumed multivariate normal distribution for fj(x,y,z,t) holds. This 
assumption was tested by examining whether a trivariate normal distribution for 
(X,Y,Z) under both treatments as well as the linearity of the regression functions 
Ej(Tlx), Ej(Tly), and Ej(Tlx,y) did hold against the data. The trivariate normal 
distribution for (X,Y,Z) under treatment j was tested using a Chi-square test by 
partitioning the sample space into 20 intervals of (x,y,z) observations (df = 
20-9-1). Furthermore, the null hypotheses of "no linear relation" for the 
regression functions Ej(Zlx) = Ej(Tlx) and Ej(Zly) = Ej(Tly) were tested for a 
usual t-test (df = n-2), with \h denoting the number of students in the sample 
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assigned to treatment j. Finally, the null hypotheses of "no linear relation" for the 
regression functions Ej(Zlx,y) = Ej(Tlx,y) were tested using the standard F-test 
(df = [nMj-m-l]), with m ss 2 denoting the number of explanatory variables. All 
p-values showed a satisfactory fit (a = 0.05). 



Utility Functions for the Separate Decisions 

For the separate selection and placement decisions, it will be assumed that the 
utility functions can be represented as linear functions of the criterion variable T 
(Mellenberg & van der Linden, 1981; van der Linden & Mellenbergh, 1977): 



Uj (t) = 



K (sL t , .(s) 
b Q (t c -t) + d Q 

u (s), . As) 
bj (t-t c ) + dj 



for i * 0 
for i = 1 , 



(16) 



b^t^O.d^ 
b?Vt c ) « df* 



for j = 0 
for j = 1 , 



(17) 



where bj (s \ bj ( ^ > 0 and [bj^-bg^l > 0 (i,j = 0,1). The condition bj (s) , b Q (s) 
> 0 states that utility must be an increasing function of the criterion for the 
acceptance decision, but a decreasing function for the rejection decision. 
Furthermore, the condition b(fP\ > 0 implies that both for assigning 

students to treatment 0 and 1, utility is an increasing function of t. Finally, the 
condition [bj^-bQ^J > 0 implies that Uq^(0 must be a more slowly 
increasing function in t than U|^(t). This condition is needed to guarantee a 
monotone solution for the separate placement decision (cf. van der Linden, 1981). 
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For the separate mastery decision the well-known threshold utility 
function is assumed (e.g., Hambleton & Novick, 1973; Huynh t 1976; Novick & 
Lindley, 1978), which is defined by the following four constants: 

for t < t c and k = 0 

for t < t n and k = 1 

c (18) 
for t > t c and k = 0 

for t > t c and k = 1, 

where dj Q < 6qq and (Iqj < djj. The condtions d 1Q < (Jqq and (Iqj < d^ 
express the assumptions that incorrect decisions represent a smaller utility than 
correct decisions. 

Monotonicity Condition s 

It should always be checked whether die conditions for weak monotonicity are 
satisfied. Doing so, it turned out that the left-hand sides of the inequalities in 
(8)-(9), (10), (11), and (12) were increasing in z for all x and y, increasing in y 
for all x and z, decreasing in y for all x t and increasing in x for all y and z t 
respectively, with some minor exceptions at the lower ends of the range of test 
scores (0 < x,y,z < 100). 

Finally, it remained still to be tested if the additional conditions on 
fj(x,y,z,t) for strong monotone solutions were met. First, the conditions of rj(t!x,y) 
and nij(zlx,y) being independent of X = x were tested comparing the linear 
regression functions Ej(T!x,y) and Ej(Tly). The null hypothesis "the variable X 
docs not deliver a significant contribution to the explanation of T" was tested 
with an F-test (df = ll,n r 3J). Second, the condition of Vj(llx,y,z) being 
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independent of X = x and Y = y ws tested comparing the linear regression 
functions Ej(T | x,y,z) and Ej(T| z). The null hypothesis "the variables X and Y 
do not deliver a significant contribution to the explanation of T" was tested using 
an F-test with df = [2,nj-4]. All p- values did not show a satisfactory fit to the test 
data, however, implying that the optimal rules did not take a strong monotone 
form (a = 0.05). Therefore, only SMMEU rules and no optimal strong rules were 
considered. 

An absolute maximum appeared to exist for the expected utility in the 
subclass of strong monotone rules, because the Hessian matrix was negative 
definite for all nonnegative test scores. 

Results for both Simultaneous and Separate Rules 

To illustrate the dependence of the results on the chosen utility structures, the 
SMMEU and the set of weak cutting scores (x c , y c (x c ), z c j(* c ,y c (x c ))) were 
computed for three different values of the utility parameters as well as for wj - 
0.6 and w 2 = 0.2, Wj = 0.1 and w 2 = 0.8, and wj = w 2 = 0.333 using the 
program NEWTON. The results are displayed in Table 2. The optimal cutting 
scores x c , y c , and l Q for the separate selection, placement, and mastery 
decisions are also reported in Table 2. It should be noted that the weak and 
SMMEU cutting scores y cr (x c ) and y* cr stayed nearly constant at approximately 
39.48 and 39.89, respectively. These cutting scores are not displayed in Table 2, 
however, because they were only used to compute the optimal rules but arc not 
used for taking decisions in an existing ISS. 
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Insert Table 2 about here 



Since 0/3x)z c j(x,y) = -PTXj.YZ / ( J TZj.XY' il followed *at z c o(x.y). 
and z cl (x t y) decreased by 1.85 and 1.05 per unit increase in x for y < y c (x) and 
y > y c (x), respectively. Similarly, (3/3y)z c j(x,y) = -PTYj.Xz'^TZj.XY im P lied 
that z c Q( x »y) and z cl (x,y) decreased by 2.05 and 1.32 per unit increase in y for y 
< y c (x) and y > y c (x), respectively. The behavior of y c (x) was examined using 
the program NEWTON. The results of the computations indicated that y c (x) was 
decreasing in x for x > x c . Hence, our expectations of the functions y c (x) and 
z c j(x,y) being decreasing in x and in both x and y for x > x c could be confirmed 
(j = 0,1). Furthermore, to illustrate the combined effect of both x and y c (x) on 
z c j(y y c (x)), the graphical displays of y c (x), z c0 (x,y c (x)), and z cl (x,y c (x)) for x 
> x c are shown in Figure 2 for utility structure (2). 



Insert Figure 2 about here 



As Figure 2 shows, y c (x), z cQ (x,y c (x)), and z cl (x,y c (x)) decreased with 
approximately 0.5, 0.8, and 0.36 per unit increase in x for x > x c . Apparently, the 
decreasing character of z c j(x,y) in x does have a stronger influence on 
z c j(x,y c (x)) than the combined effect of the decreasing character of both z c j(x,y) 
in y and y c (x) in x for utility structure (2). For all other utility structures, the 
same pattern could be observed in this study. 
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Inspection of Table 2 shows that z c Q(x c >y c (x c )) is larger than 
z c j(x c ,y c (x c )). Combined with the fact that both z c o( x >y) (when y < y c (x)) and 
z c j(x,y) (when y > y (*■)) are decreasing functions in y, it follows that students 
just accepted for the initial treatment and assigned to treatment 0 are always 
confronted with higher weak cutting scores on the mastery test than students just 
accepted for the initial treatment and assigned to treaunent 1. 

As can be seen from Table 2, X c did not show large differences 
compared to x c . However, in particular for Wj = 0.6 and W2 = 0.2, y c andz c 
were substantially higher and lower compared to y c (x c ) and z c j(* c ,y c (x c )), 
respectively (j =0,1). 

Obviously, students with selection scores X = x c were sooner assigned 
to the * higher* treatment in the case of a weak monotone approach but had to 
compensate their relatively low weak cutting scores on the placement test with 
higher optimal weak cutting scores on the mastery test. However, the decreasing 
character of z c j(x,y c (x)) in x for x > x c implies that for students with selection 
scores far above x c these rather high weak cutting scores on the mastery test 
under treatment j decreased. Also, students assigned to the 'higher' treatment 
with selection and placement scores far above x c and y c (x) needed only low 
scores on the mastery test. 

Comparison of the Expected Utilities 

As earlier noted, one might expect that in a case with empirical data the expected 
utility for a simultaneous approach will be larger than lor a separate approach. 
This expectation will now be examined comparing the expected utilities for the 
optimal weak monotone and SMMEU rules with the weighted sum of the 
expected utilities for the optimal separate rules using the program UTILITY. The 
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results are also reported in Table 2. 

As can be seen from Table 2, the expected utilities for both the optimal 
weak monotone and SMMEU rules yielded larger values than for the weighted 
sum of the expected utilities for the optimal separate rules for all nhe utility 
structures. Some utility structures, such as, for instance, utility structures (3), (6), 
and (9), even showed substantial gains in expected utility for a simultaneous 
approach. 

Furthermore, inspection of Table 2 shows that for all three approaches, 
the expected utility yielded the largest value for W2 = 0.8 in this study. In other 
words, the utility for the placement decision contributed most to the expected 
utility in our example. 

Also, comparing utility structures (l)-(3) with (4)-(6), it can be 
concluded from Table 2 that raising the utilities for the correct mastery decisions 
resulted in an increase of the expected utilities for all three approaches. 
Obviously, correct mastery decisions have a relatively strong positive influence 
on the specification of the utility structure. 

Finally, the expected utilities for the optimal weak monotone and 
SMMEU rules were compared to each other. As Table 2 shows, the expected 
utilities for the optimal weak monotone rules were larger than for the SMMEU 
rules. This result does not contradict our predictions, because the expected utility 
for an optimal weak monotone rule must yield the largest expected utility of all 
simultaneous rules if the conditions for weak monotonicity are met. 
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Discussion 

In this paper, culling scores for a selection-placemcnt-mastery problem were 
optimized simultaneously using Bayesian decision theory. The optimal decision 
procedures were illustrated empirically using data from the area of instructional 
decision making. It turned out that in some cases considerable gains in expected 
utility could be achieved by the optimal weak monotone rules compared to the 
weighted sum of the expected utilities for the optimal separate rules. 

The results indicated that the optimal weak monotone rules y c (x) and 
z r ;(x,y) were decreasing in x and both in x and y. In Vos (1994b) it is shown 
that under the same rather mild conditions on the test score distributions and 
utility functions which guarantee a weak monotone solution, optimal weak 
monotone rules are always compensatory by nature. The title of the paper already 
alludes to this result. As already explained, this feature introduces an element of 
compensation in the decision procedure: It is possible, for instance, to 
compensate low scores on the placement test by high scores on the selection test. 

A final note is appropriate. Although instructional decision making is a 
useful application of simultaneous decision making, the models advocated in this 
paper, however, are not limited to this area. Other useful applications may be 
found in such areas as psychotherapy in which it can be expected that accepted 
patients for the program react differentially to a certain kind of therapy followed 
by a success criterion, which has to be passed before being dismissed from the 
therapy. 
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Figure Caption 

Figure 1. A system of a selection, a placement, and a mastery decision 

(Case of two treatments). 
Figure 2. Graphical displays of y c (x), z c0 (x,y c (x)), and z cl (x,y c (x)) for 

x > x c in utility structure (2). 
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